For the Windows workflow run, test test\MSVC\msvc.py fails due to the precompiled headers speed test.

In #4534, I removed the multiplier (i.e., effectively 1.0):

  # TODO: Reevaluate if having this part of the test makes sense any longer
  # using precompiled headers should be faster
- limit = slow*0.90
+ limit = slow
  if fast >= limit:
      print("Using precompiled headers was not fast enough:")
      print("slow.obj:  %.3fs" % slow)

In hindsight, it probably should have been changed to slow*1.0 (or even marginally larger than 1).

This test fails frequently for me when running tests in virtual machines. Even with the change, it still occasionally fails in one vm (i.e., pre-compiled headers slower).

There are some mysterious fails due to adding the Windows runner, I guess I should have listed them here.

Failed the following 39 tests:
    test\CPPDEFINES\pkg-config.py
    test\Interactive\added-include.py and 25 others (of 28 Interactive total)
    test\MSVC\msvc.py
    test\packaging\msi\explicit-target.py
    test\packaging\msi\file-placement.py
    test\packaging\msi\package.py
    test\packaging\tar\xz_packaging.py
    test\scons-time\run\config\python.py
    test\scons-time\run\option\python.py
    test\scons-time\run\option\quiet.py
    test\scons-time\run\option\verbose.py
    test\sconsign\script\no-SConsignFile.py
    test\sconsign\script\SConsignFile.py
    test\sconsign\script\Signatures.py

Many of these are mystifying. The sconsign tests fail because the third field of entries for files is -1, which doesn't match the regex expressions \d+. But why is it getting -1?

The scons-time test - the python.py ones fail by missing the "command echo" from the files, the other two fail because we get back a path that's gone through 8+3 shortening, like:

< 'SConstruct file directory: C:\\\\Users\\\\runneradmin\\\\AppData\\\\Local\\\\Temp\\\\scons\\-time\\-[^\\\\]*\\\\foo'
---
> 'SConstruct file directory: C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\scons-time-pim4q763\\foo'

The packaging tests fail with an error I've never seen,

scons: *** Missing Packagetag 'X_MSI_LANGUAGE' for SCons.Tool.packaging.msi packager

... even though the packaging tool itself sets that.

The pkg-config test fails because perl is not found, but it's also not used !?!?!? And don't know where it found that path for pkg-config, it should be unable to look in such a weird path.

STDERR =========================================================================
'perl' is not recognized as an internal or external command,
operable program or batch file.
OSError: ' "C:/Strawberry/perl/bin/pkg-config.BAT" --cflags bug.pc' exited 9009:

There are a few others. Not able to spend time chasing these down at the moment.

The pkg-config test fails because perl is not found, but it's also not used !?!?!? And don't know where it found that path for pkg-config, it should be unable to look in such a weird path.

The pkg-config.py test contains

pkg_config_path = test.where_is('pkg-config')

On windows, when using the default pathext argument(i.e., pathext=None) for the where_is function invocation, all path extensions are tested with the basename. It appears that the Strawberry Perl bin folder contains a pkg-config.BAT which must be on the system path.

I could be wrong...

On windows, when using the default pathext argument(i.e., pathext=None) for the where_is function invocation, all path extensions are tested with the basename. It appears that the Strawberry Perl bin folder contains a pkg-config.BAT which must be on the system path.

I could be wrong...

Unlikely to be wrong. Yes, the tests are sometimes more ambitious than SCons is going to be with its limited PATH in the execution environment. I still don't know why perl gets involved.

I've pushed a change to this PR which skips the 39 tests that failed last time, which has gotten us a "pass" this time. I'll record those in an issue and we can whack those off (and reduce the skip list) as motivation appears - no rush, really.

The following could be wrong...

I still don't know why perl gets involved.

The where-is call is returning the fully qualified path to the Strawberry Perl batch file pkg-config.bat. The Strawberry Perl batch file pkg-config.bat calls perl.

pkg-config.bat:

@rem = '--*-Perl-*--
@set "ErrorLevel="
@if "%OS%" == "Windows_NT" @goto WinNT
@perl -x -S "%0" %1 %2 %3 %4 %5 %6 %7 %8 %9
@set ErrorLevel=%ErrorLevel%
@goto endofperl
:WinNT
@perl -x -S %0 %*
@set ErrorLevel=%ErrorLevel%
@if NOT "%COMSPEC%" == "%SystemRoot%\system32\cmd.exe" @goto endofperl
@if %ErrorLevel% == 9009 @echo You do not have Perl in your PATH.
@goto endofperl
@rem ';
#!/usr/bin/perl
#line 16
...

If the fully-qualified pkg-config.bat call is made in a restricted SCons environment, perl won't be on the path.

Never heard of Strawberry Perl, so I didn't realize it was a "thing", just thought the CI image had some weird paths. But it is a Thing indeed... http://www.strawberryperl.com/.

So yeah, we found something in the more liberal test framework search, and then the necessary path wasn't in env['ENV']['PATH'] when scons fired off the test. Not entirely sure how to fix that, but there will be a way...

Problems found with thie Windows runner moved to #4539